AIR QUALITY PREDICTION¶
Problem Statement: To check the quality of air using ‘Air Quality Chemical Multisensor Device’ by finding the R^2 score or coefficient of regression using different regression models and the best model is selected to evaluate the Air Quality.
In [1]:
# IMPORT LIBRARIES
import pandas as pd
import numpy as np
from pandas import Series, DataFrame
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline
import datetime
PRE-PROCESSING¶
In [2]:
# READ THE DATA
AQ = pd.read_excel("AirQuality.xlsx")
In [3]:
# DISPLAY THE DATA
AQ.head()
Out[3]:
In [4]:
#CHECKING SHAPE OF DATA
AQ.shape
Out[4]:
In [5]:
#INFORMATION OF DATA
AQ.info()
In [6]:
#CHECKING FOR NULL VALUES
AQ.isnull().any()
Out[6]:
In [7]:
#SETTING DATE AS AN INDEX
AQ.set_index("Date", inplace=True)
AQ.index = pd.to_datetime(AQ.index)
type(AQ.index)
Out[7]:
In [8]:
#CONVERTING HH:MM:SS INTO HH FORMAT
AQ['Time'] = pd.to_datetime(AQ['Time'],format = '%H:%M:%S').dt.hour
AQ.head(3)
Out[8]:
In [9]:
#DATA DESCRIPTION
AQ.describe()
Out[9]:
In [10]:
#COUNT OF -200 VALUE
AQ[AQ['CO(GT)']==-200].count()
Out[10]:
In [11]:
#REPLACING THE NEGATIVE VALUES.
AQ.replace(to_replace= -200, value= np.NaN, inplace= True)
In [12]:
#CHECKING NULL VALUES.
AQ.isnull().any()
Out[12]:
In [13]:
AQ.isnull().sum()
Out[13]:
REMOVING NEGATIVE AND NULL VALUES¶
In [14]:
#YELLOW COLOR IN A PLOT SHOWS THE NULL VALUE
plt.figure(figsize=(8,6))
sns.heatmap(AQ.isnull(),yticklabels=False,cbar=False,cmap='viridis')
plt.show()
In [15]:
AQ.shape
Out[15]:
In [16]:
#REPLACING NULL VALUES WITH MEDIAN
AQ.fillna(AQ.median(), inplace=True)
In [17]:
sns.heatmap(AQ.isnull(),yticklabels=False,cbar=False,cmap='viridis')
plt.show()
In [18]:
# GENERATING CORRELATION MATRIX
corr = AQ.corr()
# PRINT THE CORRELATION MATRIX
corr
Out[18]:
CORRELATION MATRIX PLOT¶
In [19]:
plt.figure(figsize=(10, 5))
# plotting the heat map
# corr: give the correlation matrix
# cmap: colour code used for plotting
# vmax: gives maximum range of values for the chart
# vmin: gives minimum range of values for the chart
# annot: prints the correlation values in the chart
# annot_kws: Sets the font size of the annotation
sns.heatmap(corr, cmap='YlGnBu', vmax=1.0, vmin=-1.0, annot = True, annot_kws={"size": 12})
# specify name of the plot
plt.title('Correlation Matrix')
plt.show()
DETECTING OUTLIERS.¶
In [20]:
# CREATE A BOXPLOT FOR ALL NUMERIC FEATURES.
plt.figure(figsize=(15, 5))
AQ.boxplot(['PT08.S1(CO)','PT08.S2(NMHC)','NOx(GT)','PT08.S3(NOx)','PT08.S4(NO2)','PT08.S5(O3)','NMHC(GT)'])
plt.show()
In [21]:
plt.figure(figsize=(15, 5))
AQ.boxplot(['CO(GT)', 'C6H6(GT)', 'T', 'RH', 'AH'])
plt.show()
EXPLORATION¶
SCATTER PLOT OF WHOLE DATA¶
In [22]:
sns.pairplot(AQ)
Out[22]:
In [23]:
#SCATTER PLOT OF SELECTED FEATURES
#SELECTED FEATURES ARE--'PT08.S1(CO)','PT08.S2(NMHC)','NOx(GT)','PT08.S3(NOx)','PT08.S4(NO2)','PT08.S5(O3)','NMHC(GT)',
#'CO(GT)', 'C6H6(GT)'
sns.set_style('whitegrid')
eda_AQ = AQ.drop(['Time','RH','AH','T'], axis=1)
sns.pairplot(eda_AQ)
Out[23]:
VISUALIZATION OF DATA¶
In [24]:
#HISTOGRAM OF EVERY FEATURE.
AQ.hist(figsize = (20,20))
plt.show()
CONCLUSION FROM HISTROGRAM : From the histogram, we can observe the variability of each attribute. Also we can observe the skewness of data.
DISTPLOT¶
The distplot shows the distribution of a univariate set of observations.
In [27]:
plt.title("True hourly averaged concentration CO in mg/m^3 Distribution")
sns.distplot(AQ['CO(GT)'])
Out[27]:
In [28]:
plt.title("PT08.S1 (tin oxide) hourly averaged sensor response (nominally CO targeted) Distribution")
sns.distplot(AQ['PT08.S1(CO)'])
Out[28]:
In [29]:
plt.title("True hourly averaged overall Non Metanic HydroCarbons concentration in microg/m^3 Distribution")
sns.distplot(AQ['PT08.S2(NMHC)'])
Out[29]:
In [30]:
plt.title("True hourly averaged Benzene concentration in microg/m^3 Distribution")
sns.distplot(AQ['C6H6(GT)'])
Out[30]:
In [31]:
sns.distplot(AQ['T'])
Out[31]:
BARPLOT¶
In [33]:
AQ['NOx(GT)'].resample('M').mean().plot(kind='bar', figsize=(18,6))
plt.xlabel('Month')
plt.ylabel('Total Nitrogen Oxides(NOx)')
plt.title("Mean Total Nitrogen Oxides (NOx) Level by Month")
Out[33]:
CONCLUSION FROM BARPLOT - We can see that initially, the Nitric Oxide levels are low but as the year pass, the Nitric Oxide level is increased.
In [35]:
plt.figure(figsize=(20,6))
sns.barplot(x='Time',y='NOx(GT)',data=AQ, ci=False)
plt.xlabel('Hours')
plt.ylabel('Total Nitrogen Oxides(NOx)')
plt.title("Mean Total Nitrogen Oxides (NOx) Frequency During Days")
Out[35]:
SCATTERPLOT¶
In [36]:
AQ.plot(x='NO2(GT)',y='NOx(GT)', kind='scatter', figsize = (10,6), alpha=0.3)
plt.xlabel('Level of Nitrogen Dioxide')
plt.ylabel('Level of Nitrogen Oxides(NOx)')
plt.title("Mean Total Nitrogen Oxides (NOx) Frequency During Days")
plt.tight_layout();
In [37]:
plt.xlabel("Temperature")
plt.ylabel('Relative Humidity')
plt.xlim(0,40)
plt.title("Relative Humidity vs Temperature-Full")
plt.scatter(AQ['T'],AQ["RH"],marker=".")
Out[37]:
In [38]:
plt.xlabel("Temperature in Degree Celsius")
plt.ylabel('Relative Humidity')
plt.xlim(-5,10) #Get or set the x limits of the current axes.
plt.title("Relative Humidity vs Temperature")
plt.scatter(AQ['T'],AQ["RH"],marker=".")
Out[38]:
PREDICTION¶
In [39]:
#IN THE DATASET,TAKING C6H6(GT) AS TARGET FEATURE
#BY OBSERVING THE TARGET VALUES WE CAN CONCLUDE THAT, IT IS OF TYPE -- NUMERICAL(REGRESSION)
#SO NEED TO USE REGRESSION ALGORITHMS
In [40]:
#SPLITING THE DATASET INTO TRAIN AND TEST
In [41]:
x = AQ.drop('C6H6(GT)',axis=1)
y = AQ['C6H6(GT)']
In [42]:
x
Out[42]:
In [43]:
y
Out[43]:
In [63]:
from sklearn.model_selection import train_test_split
x_train,x_test,y_train,y_test = train_test_split(x,y,test_size=0.2,random_state=62)
STANDARDIZATION (SCALING)¶
Standardizing a dataset involves rescaling the distribution of values so that the mean of observed values is 0 and the standard deviation is 1.
In [46]:
from sklearn.preprocessing import StandardScaler
std = StandardScaler()
std.fit(x_train)
x_train = std.transform(x_train)
x_test = std.transform(x_test)
K-NEAREST NEIGHBOUR¶
K nearest neighbors is a simple algorithm that stores all available cases and predict the numerical target based on a similarity measure (e.g., distance functions).
In [48]:
from sklearn.neighbors import KNeighborsRegressor
knn = KNeighborsRegressor(n_neighbors=5)
knn.fit(x_train,y_train)
Out[48]:
In [49]:
y_pred = knn.predict(x_test)
y_pred
Out[49]:
In [50]:
df1 = pd.DataFrame({'Actual':y_test,'Predicted':y_pred})
df1
Out[50]:
In [51]:
#VISUALISING THE RESULT
In [52]:
plt.figure(figsize=(12,6))
plt.scatter(y_test,y_pred,marker='+')
Out[52]:
R^2 SCORE¶
R-squared is a statistical measure that represents the goodness of fit of a regression model. The ideal value for r-square is 1. The closer the value of r-square to 1, the better is the model fitted.
In [54]:
from sklearn.metrics import r2_score
a=r2_score(y_test,y_pred)*100
a
Out[54]:
LINEAR REGRESSION¶
Linear regression finds the linear relationship between the dependent variable and one or more independent variables using a best-fit straight line
In [56]:
from sklearn.linear_model import LinearRegression
LR = LinearRegression()
LR.fit(x_train,y_train)
Out[56]:
In [57]:
y_pred = LR.predict(x_test)
In [58]:
y_pred
Out[58]:
In [59]:
df2 = pd.DataFrame({'Actual':y_test,'Predicted':y_pred})
df2
Out[59]:
In [60]:
#VISUALISING THE RESULT(PLOTTING BESTFIT LINE)
In [61]:
plt.figure(figsize=(6,3))
plt.scatter(y_test,y_pred,marker='+')
plt.plot([0, 50], [0, 50], '--k')
plt.axis('tight')
#'tight'-- Set limits just large enough to show all data, then disable further autoscaling.
Out[61]:
In [62]:
#Testing data
from sklearn.metrics import r2_score
b = r2_score(y_test,y_pred)*100
b
Out[62]:
DECISION TREE (REGRESSION)¶
Decision tree regression observes features of an object and trains a model in the structure of a tree to predict data in the future to produce meaningful continuous output.
In [65]:
from sklearn.tree import DecisionTreeRegressor
regressor = DecisionTreeRegressor(random_state = 0)
regressor.fit(x, y)
Out[65]:
In [66]:
y_pred=regressor.predict(x_test)
y_pred
Out[66]:
In [67]:
y1_pred=regressor.predict(x_train)
y1_pred
Out[67]:
In [68]:
df3 = pd.DataFrame({'Actual':y_test,'Predicted':y_pred})
df3
Out[68]:
In [69]:
#VISUALIZATION OF RESULT
In [70]:
plt.figure(figsize=(12,6))
plt.scatter(y_test,y_pred,marker='+')
Out[70]:
In [71]:
from sklearn.metrics import r2_score
c=r2_score(y_test,y_pred)*100
c
Out[71]:
In [72]:
print("r2_score of K-NN Regression is : ",a)
print("r2_score of linear Regression is : ",b)
print("r2_score of Decision Tree Regression is : ",c)
CONCLUSION¶
The r2_score is higher for the Decision Tree, so we can conclude that Decision Tree Regression model is best for evaluating the Quality of Air for a given Air Quality Prediction dataset